feat(box): a bulk Stop runs on the box, so closing the laptop doesn't halt it - #192

Stacked on #175 (archive in the background) and on the background-jobs tray branch (Stener1:background-jobs-tray, not opened upstream yet). Both are rebased onto current main. The diff includes their commits until they land: the tray is the six commits after #175's, and this PR's own work is the ten commits from feat(box): a bulk Stop runs on the box… on.

Summary

Closing the laptop mid bulk Stop used to halt it. The app ran the whole sequence, so in-flight and queued requests failed on wake, and agents ended up stopped but not archived. Now each box with runs does its share itself, as one durable stop run, and so does the sidebar's single Archive…:

runStopJob (app)                                   box (internal/box)
  split rows by box
  keep the job (berth.jobs.stopRuns) ── before anything is sent
  box has runs + online ──► POST /v1/runs {template: stop, group, idem_key}
     no answer ─► ask again, same key, until it answers (rows: Waiting for BOX)
                                                    map  stop_session ×N   (3 at a time, session.stop gate)
                                                    map  archive_worktree  (one at a time)
                                                           archiveTurn: one archive at a time on the box, across runs
                                                           live agent there?   → Kept: …
                                                           uncommitted work?   → Kept: uncommitted changes
                                                           worktree.remove gate
                                                           dropWorktreeIf(no force, branch kept)
                                                             archive script, then under lastLook:
                                                             re-check agents, remove (no session starts in between)
  follow GET /v1/runs/{id} ──► rows                (no answer = ask again; only the box fails a row)
  "Stop the rest" ──► cancel the run; a start still unanswered is looked up by group (GET /v1/runs?group=)
                      and cancelled if the box took it; the cancel is kept across a relaunch
  else: the app's own loop (older boxes, offline at start, a 4xx on start),
        each box's line held until the box says an archive script finished

Archive… (sidebar, zen, launcher)
  box has runs ──► one-worktree stop run: its sessions stop, then archive_worktree as above
  else         ──► the app's per-box line, held until worktree.removed / worktree.archive.failed

• Box: the stop template and stop_session / archive_worktree leaves (runstop.go). dropWorktree gets an in-flight guard, so a second ask (a resumed run, a duplicate run, a second DELETE) joins the first instead of running the archive script again. The dirty check comes before that guard, so a second ask for a dirty worktree is refused, not answered 202. archiveTurn serializes archives across a box's stop runs. removeAfter / startUnseen close the check-then-remove race: no session starts between the last look for agents and the folder going. If berthd stops mid-archive, the step stays unfinished in the journal for the resumed run.
• App: lib/stop-job.ts keeps the job before it sends, asks an unanswered box again under one key until it answers, and looks a run up by its group when Stop the rest comes first. A job comes back to the tray after a relaunch (for at most 24 h), cancelled if it was. The Worktrees table's Stop sessions and Archive, the sidebar's Stop everything in … and its Archive… use the same run (lib/stop-on-box.ts, archiveOnBox).
• The mock plays stop runs out against its own fixtures (playStop), in each box's archive line. ?noruns gives the old path.
• An archive that fails says why, and one stopped halfway can be finished (fix(box): an archive that fails says why…). git worktree remove ran under the 60 s limit that every git call has. On a busy box a big worktree took longer: git was killed mid-delete without a word, so the row read Couldn't archive: git worktree remove:, and the folder was left without its .git, which git then refused to remove again (validation failed … .git does not exist). Now:
  • removal gets 10 minutes (BERTH_WORKTREE_REMOVE_TIMEOUT sets another limit), and a failure with no output says how it ended: timed out, stopped by a signal, cancelled, or its exit status (gitFailed);
  • a worktree git lists as prunable is finished on archive (finishRemoval). What's left of its folder goes, then git's record of that one worktree, never git worktree prune, which would drop every half-removed worktree's record and orphan their folders. The branch is kept. It only does this when git, asked through the record's HEAD and index, sees nothing but files the removal already deleted, and no repository is inside the folder. Any .gitignore the removal deleted is put back as committed first, since otherwise node_modules reads as new files; otherwise it's Kept: uncommitted changes, and Remove anyway still forces. A worktree whose folder is gone is pruned the same way. Its archive script isn't run again;
  • whether a worktree still has its .git is checked each time the list is read, not taken from git's prunable line, so a cached listing (fix(box): listing projects runs git only when a repository changed #195) can't send such an archive back to git worktree remove;
  • a removal's last look for agents now holds only its own folder (lookGuard, which replaces the box-wide lastLook RWMutex), so a removal that takes minutes doesn't hold up session starts anywhere else on the box.
• Docs: the stop template in guides/runs.mdx, the new step kinds, changelog bullets, and the verify skill's feature map and scenarios.

Evidence

All on the final head, with the verify-berthd skill.

A removal git doesn't finish in time (new dashboard-stop-interrupted-live, a real dev box, synthetic acme-shop). The box is restarted with a git on its PATH that, for verify-slow only, deletes the worktree's .git and .gitignore and then hangs (with a node_modules only that .gitignore ignored), as a big worktree's git worktree remove on a loaded box did. Then verify-slow and verify-ok are bulk-stopped with archiving ticked.

• Before (VERIFY_BERTHD= this branch's previous head): the archive step failed after 1m0.283s, the 60 s git limit, with the error "git worktree remove: ". The tray row ended at the colon. git worktree list showed verify-slow as prunable with its files still there, and archiving it again failed for good: git worktree remove: fatal: validation failed, cannot remove working tree: '…/acme-shop-verify-slow/.git' does not exist.
• After (BERTH_WORKTREE_REMOVE_TIMEOUT=5s):

- the failed row reads: verify-slow Stopped · Couldn't archive: git worktree remove timed out after 5s and was stopped; the folder may be partly removed, and archiving it again finishes it
- git agrees: verify-ok is gone, verify-slow is half removed (prunable, its files still there)
- Archive… verify-slow from the sidebar finishes it
- its folder is gone, git no longer lists it, and both branches stay

Before: a bare colon	After: the cause	After: Archive… finishes it

The scenario allows the console's 404 for GET sessions/<stopped agent>/screen: the board's card asks for a just-stopped agent's screen once. That happens without this change too, and each failed request is logged as a step.

A start the box never answers (new dashboard-stop-unanswered, a real dev box, synthetic acme-shop). berthd is killed (SIGKILL) as Stop is pressed and started again 20 s later.

• Before: startStopRun gave up after three asks (about 6 s) and failed the rows, Failed: devq is offline, though the box might have the run. That was the old unit test a box that never answers the start fails its rows.
• After: no row fails, the start is asked again once the box is back, and the box ends with one run for the job:

- with the box down the rows say: verify-mute-gp4k-a Waiting for devq | verify-mute-gp4k-b Waiting for devq
- after 20 s down, no row failed
- one stop run on the box for it (group stop-mvqsgss9yzpz): r_mvqshg2697ke succeeded
- git on the box agrees: both worktrees gone, both branches kept

Box down: rows wait	Box back: done

Bulk Stop and sidebar Archive… on a real box (dashboard-stop-live): the report reads 3 stopped, 2 archived, 1 kept; git agrees; the box lists Stopping 3 agents: succeeded and Archive verify-solo: succeeded. No request is refused any more: the dirty worktree is kept inside the run.

Report	Archive… as a stop run

The lid closing (dashboard-stop-lid): the laptop agent was paused for 25 s mid-archive, and on wake the tray read 5 stopped, 3 archived, 2 kept. A reload mid-run brought the job back, ending 2 stopped, 2 archived. The Worktrees table's Stop sessions and Archive each ran as a stop run.

After waking	Reattached, done

One archive at a time per box (mock, worktree-archive): Archive… on gpu/ci-flake, then judge-v2 at once. Both are stop runs on gpu, and judge-v2 waits in gpu's line (Archived ci-flake at 5.5 s, Archived judge-v2 at 10.0 s). qa-deck's row stays Archiving… until its script is done, then shows one toast.

Both archiving	Script running	Dirty: kept

Tests:

• Go, new:
  • TestStopRunsOnABoxArchiveOneAtATime: two runs' scripts log start end start end.
  • TestNoAgentStartsUnseenAsAWorktreeGoes: a session asked for during the last look waits, then gets no folder.
  • TestAWorktreeAskedAgainIsCheckedForWorkFirst: dirty while the first archive runs, so the second DELETE is refused.
  • The dedup test now waits for worktree.archive.started and quotes its path.
• Go, new for the interrupted removal:
  • TestAWorktreeRemoveThatFailsWithNoOutputSaysWhy and TestAWorktreeRemoveThatTimesOutSaysSo, with a fake git on PATH; TestAStopRunSaysWhyARemovalFailed through a stop run.
  • TestAHalfRemovedWorktreeIsFinished: folder and record gone, branch kept, another half-removed worktree left alone; a live dirty worktree and the main checkout are still refused.
  • TestAHalfRemovedWorktreeWithWorkInItIsKept (a changed or an added file: kept in git's words; Remove anyway deletes it) and TestAHalfRemovedWorktreeHoldingARepositoryIsKept.
  • TestAWorktreeWhoseFolderIsGoneIsPruned (a worktree can be made at that path again) and TestAWorktreeGitStillHasIsNeverDeletedByHand.
  • TestAStopRunFinishesAHalfRemovedWorktree: half-removed → Archived, folder gone → already gone and pruned, one with work in it → Kept: uncommitted changes; the archive script never runs again.
  • TestAHalfRemovedWorktreeThatLostItsGitignore: .gitignore files deleted before node_modules → finished; with a really new file as well → kept.
  • TestAListingFromBeforeTheGitWentShowsItDropped: a listing read before the .git went still shows the worktree as dropped.
  • TestNoAgentStartsUnseenAsAWorktreeGoes now also starts a session in another worktree during the last look.
• Unit, new: kept-reason (a timeout, a signal or a silent failure is a failure, not kept) and stop-job (the box's words reach the row unchanged).
• Go, the full suite: go vet ./... && go test -race ./... fails only in tests that fail on main on this machine under load (TestTheCatalogIsCapped, TestReviewRoutesOverTheWire, TestDetach, TestServicesAreCounted); TestReviewRoutesOverTheWire passes 3/3 alone. On the interrupted-removal head it failed only TestDetach (it fails on main's tree too). Under load, earlier runs also failed TestForwardReachesTheBoxAndSurvivesAnAgentRestart, TestATeamSetupResumesAfterBerthdRestarts and TestSleepDurable; each passes 3/3 alone, on this branch and on its previous head.
• Unit, new in stop-job.test.ts:
  • an unanswered start is asked until answered
  • Stop the rest while unanswered: a run the box took is cancelled, and with none nothing is done
  • kept before sending, then asked again after a relaunch
  • a cancel while out of reach is kept: after a relaunch it is only looked up and cancelled, never started (both with and without the box having taken it)
  • a 404'd run clears its Archiving… marks
  • an archive script holds its box's line
• Unit, elsewhere: runPool with an undefined item, and unmarked. pnpm test: 636 pass.
• E2E: dashboard-stop-archive.spec.ts and worktree-remove.spec.ts pass, 11/11.
• These pass: typecheck:plugins, check:titles, check:type, check:grid, check:csp, check:themes, tsc -p e2e and pnpm build. cargo check was not run: GTK dev libs are missing on this machine, which is unrelated.
• Codex review and adversarial review ran on the interrupted-removal change. They found three problems, all fixed in its commit: a remnant's files were deleted without checking them for work, a retry re-ran the archive script on a half-deleted folder, and another repository nested under an ignored folder could be deleted with it.
• The 29 review threads from cubic are each answered with the commit that addresses them. Codex review and adversarial review ran on the fixes; their findings (a cancel lost across a relaunch, and an archive failure that beats the 202) are fixed in the last commit.

Merge Danger

Door: two-way

On the box it's additive: a new template, two step kinds, an in-flight guard, an archive line across stop runs, and a lock held only from a removal's last look for agents until the folder goes. Old clients are unaffected, and new clients fall back on boxes without runs. Reverting the PR restores the app's loop.

Blast Radius: bulk-stop and archive

• Where it reaches: on boxes with runs, the board's bulk Stop / Clean up, the Worktrees table's Stop sessions and Archive, the sidebar's Stop everything and its single Archive… all go through the box.
• What changes there:
  • A row reads Archived once the script is done, not Archiving while it runs.
  • Archive stops the worktree's sessions first, as archiving always ended them.
  • A worktree is kept if an agent starts there during the script.
  • A bulk Stop row waits for an unreachable box instead of failing.
  • A session start in a worktree being removed waits for that removal; starts elsewhere don't wait.
  • git worktree remove may take up to 10 minutes instead of 60 s, and a worktree git dropped the link to is finished by the box itself (its folder, then its record under .git/worktrees/), only when nothing in it is work. Branches are always kept.
• On boxes without runs: the app holds a box's line until an archive script finishes, so queued archives there wait longer, but never overlap.

🤖 Generated with Claude Code